Bidirectional optical flow estimation method and apparatus

ABSTRACT

A bidirectional optical flow estimation method and apparatus are provided. The method includes acquiring a target image pair of which optical flow is to be estimated, and constructing an image pyramid for each target image in the target image pair respectively, and performing bidirectional optical flow estimation using a pre-trained optical flow estimation model based on the image pyramid, to obtain bidirectional optical flow between the target images. An optical flow estimation module in the optical flow estimation model is recursively called to perform the bidirectional optical flow estimation sequentially based on images of respective layers in the image pyramid according to a preset order, forward warping towards middle processing is performed on an image of a corresponding layer of the image pyramid before each call of the optical flow estimation module, and an image of an intermediate frame obtained by the forward warping towards middle processing is inputted into the optical flow estimation module. With the disclosure, the efficiency and generalization of bidirectional optical flow estimation can be improved, and model training and optical flow estimation overheads can be reduced.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. §119(a) of a Chinese patent application number 202210207725.2, filed onMar. 4, 2022, in the Chinese Intellectual Property Office, thedisclosure of which is incorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to computer vision technology. More particularly,the disclosure relates to a bidirectional optical flow estimation methodand apparatus.

2. Description of Related Art

In computer vision, optical flow is often used to characterizepixel-level motions in images, which may be caused by camera movement orby the motion of an object. The optical flow (also called an opticalflow field) refers to a set of pixel displacements between two adjacentframes of pictures, i.e. a set of displacement vectors generated in aprocess of moving each pixel in the previous picture to a correspondingpixel position in the subsequent picture. Optical flow estimation is aclassical problem in computer vision, or a key step of many videounderstanding algorithms. Video frame interpolation, moving objectdetection, video content understanding, and other algorithms often relyon accurate optical flow information.

The optical flow may be divided into sparse optical flow and denseoptical flow according to whether to select image sparse points foroptical flow estimation. The dense optical flow describes optical flowof each pixel of an image moving to a next frame. The optical flow in ageneral context refers to the dense optical flow, and the disclosurealso proposes a technical solution for the dense optical flow.

An optical flow estimation method based on feature pyramids is acommonly used optical flow estimation algorithm. FIG. 1 is a schematicdiagram of optical flow estimation of the method according to therelated art.

Referring to FIG. 1 , according to the method, feature pyramids (i.e.feature pyramid 1 and feature pyramid 2 in the figure) are constructedbased on two adjacent frames of original input pictures respectively. Asthe number of pyramid layers is increased, a feature size is graduallyreduced. Then, feature data of the 0th layer of the feature pyramid isinputted into an optical flow estimation model for processing to obtainan optical flow estimation value. The optical flow estimation model is aconvolutional neural network (CNN) composed of a warping layer, a costvolume layer and an optical flow estimation layer.

During the implementation of the disclosure, the inventors have foundthat the above-mentioned existing optical flow estimation method hasproblems such as large operational overheads, low efficiency and poorgeneralization.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

By research and analysis, it is found that the reasons for theabove-mentioned problems are as follows.

In the above-mentioned optical flow estimation method, the optical flowestimation model has a large scale of parameters, so that the opticalflow estimation model has large training overheads and low operationalefficiency.

The robustness of the optical flow estimation model in theabove-mentioned optical flow estimation method is limited by the picturescale of a target data set during model training. When a scale requiredfor optical flow by a downstream task interfacing with the optical flowestimation method is greater than the picture scale of the training dataset, the corresponding optical flow estimation cannot be performed basedon the optical flow estimation model. Therefore, the robustness of theabove-mentioned optical flow estimation method for different scales ofoptical flow is limited by the picture scale of the target data setduring training, and it is often unable to achieve good generalizationresults in practical applications.

The above-mentioned optical flow estimation method can only obtainunidirectional optical flow between adjacent frames when running once,and needs to run twice for bidirectional optical flow. Therefore,estimation for the bidirectional optical flow is low in efficiency andcannot meet the requirements of real-time performance.

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providea bidirectional optical flow estimation method and apparatus, which canimprove the efficiency and generalization of bidirectional optical flowestimation and reduce model training and optical flow estimationoverheads.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, a bidirectional opticalflow estimation method is provided. The method includes acquiring atarget image pair of which optical flow is to be estimated, andconstructing an image pyramid for each target image in the target imagepair, performing bidirectional optical flow estimation using apre-trained optical flow estimation model based on the image pyramid foreach target image to obtain bidirectional optical flow between thetarget images of the target image pair, wherein an optical flowestimation module in the optical flow estimation model is recursivelycalled to perform the bidirectional optical flow estimation sequentiallybased on images of respective layers in the image pyramid for eachtarget image according to a preset order, wherein forward warpingtowards middle processing is performed on an image of a correspondinglayer of the image pyramid before each call of the optical flowestimation module, and wherein an image of an intermediate frameobtained by the forward warping towards middle processing is inputtedinto the optical flow estimation module.

In accordance with another aspect of the disclosure, a bidirectionaloptical flow estimation apparatus is provided. The apparatus includes animage pyramid construction unit, configured to acquire a target imagepair of which optical flow is to be estimated, and construct an imagepyramid for each target image in the target image pair, and an opticalflow estimation unit, configured to perform bidirectional optical flowestimation using a pre-trained optical flow estimation model based onthe image pyramid for each target image to obtain bidirectional opticalflow between the target images of the target image pair, wherein anoptical flow estimation module in the optical flow estimation model isrecursively called to perform the bidirectional optical flow estimationsequentially based on images of respective layers in the image pyramidaccording to a preset order, wherein forward warping towards middleprocessing is performed on an image of a corresponding layer of theimage pyramid before each call of the optical flow estimation module,and wherein an image of an intermediate frame obtained by the forwardwarping towards middle processing is inputted into the optical flowestimation module.

Embodiments of the disclosure also provide a bidirectional optical flowestimation device, including a processor and a memory.

The memory stores an application program executable by the processor forcausing the processor to perform the bidirectional optical flowestimation method as described above.

Embodiments of the disclosure also provide a computer-readable storagemedium, storing computer-readable instructions for performing thebidirectional optical flow estimation method as described above.

Embodiments of the disclosure also provide a computer program product,including computer programs/instructions which, when executed by aprocessor, implement the steps of the bidirectional optical flowestimation method as described above.

In summary, according to the bidirectional optical flow estimationscheme proposed in the disclosure, bidirectional optical flow estimationis performed in a recursive calling manner based on the image pyramidsfor a target image pair of which optical flow is to be estimated. Thus,on the one hand, the speed of optical flow estimation can be increasedby using the image pyramids, and on the other hand, the number ofparameters of the model can be reduced by recursive calling, and modeltraining and optical flow estimation overheads can be further reduced.Moreover, by combining the recursive calls and the image pyramids, therobustness for different scales of optical flow can be improved and thegeneralization can be enhanced. In addition, before each optical flowestimation, forward warping towards middle processing is performed onimages of corresponding layers of the image pyramid, and optical flowestimation is performed based on an image of an intermediate frameobtained by the processing. Thus, the accuracy of optical flowestimation can be improved. Therefore, with the embodiments of thedisclosure, the efficiency, accuracy and generalization of bidirectionaloptical flow estimation can be improved, and model training and opticalflow estimation overheads can be reduced.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a schematic diagram of an existing optical flow estimationmethod based on feature pyramids according to the related art;

FIG. 2 is a schematic flowchart of a method according to an embodimentof the disclosure;

FIG. 3 is a schematic diagram of a process of optical flow estimationbased on an example of an optical flow estimation model according to anembodiment of the disclosure;

FIG. 4 is a schematic diagram of optical flow estimation of an opticalflow estimation module according to an embodiment of the disclosure;

FIGS. 5, 6, 7, and 8 show schematic diagrams of embodiments applied todifferent application scenarios according to various embodiments of thedisclosure; and

FIG. 9 is a structural schematic diagram of an apparatus according to anembodiment of the disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

FIG. 2 is a schematic flowchart of a bidirectional optical flowestimation method according to an embodiment of the disclosure.Referring to FIG. 2 , the bidirectional optical flow estimation methodimplemented in this embodiment includes the following operations201-202.

In operation 201, a target image pair of which optical flow is to beestimated is acquired, and an image pyramid is constructed for eachtarget image in the target image pair respectively.

In this operation, in order to facilitate improving the efficiency ofsubsequent optical flow estimation, a corresponding image pyramidinstead of a feature pyramid is constructed respectively based on eachimage in a target image pair (specifically composed of images of twosuccessive frames) of which optical flow is to be estimated currently.In this way, in a subsequent operation, forward warping towards middleprocessing (i.e. a forward warping operation) may be quickly performedbased on the images, so that recursive optical flow estimation may beperformed using a result of middle forward warping processing, and theefficiency and accuracy of bidirectional optical flow estimation can befurther improved.

The construction of the image pyramid in the operation may be achievedusing existing methods. The specific number of layers of the imagepyramid is related to the scale of a target image, and the larger thescale of a target image has, the more the number of layers of thepyramid has.

In practical applications, a three-layer image pyramid may beconstructed, i.e. processing an original target image into threepictures with different scales. The width and height of the uppermostimage are both ¼ of those of the original image, the width and height ofthe middle image are ½ of those of the original image, and the lowermostimage is the original image.

In operation 202, bidirectional optical flow estimation is performedusing a pre-trained optical flow estimation model based on the imagepyramid, to obtain bidirectional optical flow between the target images.An optical flow estimation module in the optical flow estimation modelis recursively called to perform the bidirectional optical flowestimation sequentially based on images of respective layers in theimage pyramid according to a preset order, forward warping towardsmiddle processing is performed on an image of a corresponding layer ofthe image pyramid before each call of the optical flow estimationmodule, and an image of an intermediate frame obtained by the forwardwarping towards middle processing is inputted into the optical flowestimation module.

The operation is used for performing the bidirectional optical flowestimation using a pre-trained optical flow estimation model based onthe image pyramids for two target images obtained in operation 201.

Here, when the bidirectional optical flow estimation is performed, theoptical flow estimation module in the optical flow estimation modelneeds to be recursively called to perform the bidirectional optical flowestimation. That is, each time the optical flow estimation module iscalled to perform the bidirectional optical flow estimation, an opticalflow estimation result outputted from a previous call of the opticalflow estimation module is needed to perform the optical flow estimation.In this way, all optical flow estimations may share parameters byadopting a model structure of recursively calling the optical flowestimation module. Thus, on the one hand, the number of parameters ofthe model can be greatly reduced, and on the other hand, multi-scaleoptical flow training can be facilitated. By combining the imagepyramids with the recursive optical flow estimation, the model can betrained or refined on low-resolution data, but can achieve goodgeneralization results on high-resolution pictures, and has strongrobustness for different scales of optical flow, thus effectivelyimproving the robustness of the model for different scales of opticalflow.

In addition, in the operation, before each call of the optical flowestimation module, Forward warping towards middle processing on an imageof a corresponding layer of the image pyramid (i.e. an image in theimage pyramid, which needs to be inputted into the optical flowestimation module for optical flow estimation currently) is performed.Then, the optical flow estimation module is called to perform opticalflow estimation based on an image of an intermediate frame obtained bythe forward warping towards middle processing, i.e. to perform warpingprocessing on pictures of two successive frames towards an intermediateframe based on bidirectional optical flow, so that the same object inthe successive frames is moved to a near position. In this way, on theone hand, it is advantageous to encode currently estimated optical flowon the feature level, and on the other hand, it is convenient toconstruct an accurate cost volume in the optical flow estimation module,and the cost volume is a very discriminative feature for optical flowestimation, so that the accuracy of optical flow estimation can beeffectively improved.

FIG. 3 shows a schematic diagram of a process of optical flow estimationbased on an example of an optical flow estimation model according to anembodiment of the disclosure. Referring to FIG. 3 , final bidirectionaloptical flow is obtained by three recursive calls of the optical flowestimation module in the optical flow estimation model based on eachlayer of image for two three-layer image pyramids of a target image pairsequentially. That is, starting from images with the lowest resolution,the image pyramids are iterated for three times, and the finalbidirectional optical flow is outputted on pictures with the highestresolution. Before each call of the optical flow estimation module toperform optical flow estimation, a forward warping operation on imagesof two successive frames on a corresponding pyramid layer is performed,so that an image of an intermediate frame obtained by the forwardwarping operation may be used to perform optical flow estimationaccurately and quickly.

Preferably, in order to save operational overheads and better processoptical flow with a large scale, in one implementation, the order ofperforming optical flow estimation based on the image pyramid may be: anascending order of image scales, i.e. the optical flow estimation moduleis recursively called to perform bidirectional optical flow estimationby traversing respective layers of images in the pyramid sequentiallyfrom the top of the pyramid to the bottom of the pyramid.

In one implementation, forward warping towards middle processing may beperformed specifically by the following methods.

If the optical flow estimation module is to be called for the first timecurrently, the forward warping towards middle processing is performedbased on uppermost images in the image pyramids and initialbidirectional optical flow, to obtain images of intermediate framescorresponding to the uppermost images respectively; or otherwise, theforward warping towards middle processing is performed based oncorresponding images in the image pyramids of a current call of theoptical flow estimation module and bidirectional optical flow outputtedfrom a previous call of the optical flow estimation module, to obtainimages of intermediate frames of the corresponding images respectively.The initial bidirectional optical flow is 0.

In the above-mentioned method for forward warping towards middleprocessing, the bidirectional optical flow obtained from the previousoptical flow estimation is needed to perform the forward warping towardsmiddle processing, and thus the optical flow which has been currentlyestimated is encoded on a feature level, so as to facilitate improvingthe robustness of the model for different scales of optical flow.

FIG. 4 is a schematic diagram of a process of bidirectional optical flowestimation using an optical flow estimation model according to anembodiment of the disclosure. Referring to FIG. 4 , in oneimplementation, bidirectional optical flow estimation may be performedspecifically by the following steps.

In step 401, feature extraction is performed on the image of theintermediate frame inputted to the optical flow estimation module usinga CNN feature extractor, to obtain a CNN feature of the image of theintermediate frame.

The specific method for feature extraction in the step is known to aperson skilled in the art, and detailed descriptions thereof are omittedherein.

In step 402, a corresponding cost volume is determined based on the CNNfeature of the image of the intermediate frame.

The step is used to construct a corresponding cost volume based on theCNN feature of the image of the intermediate frame, so as to use thecost volume to improve the accuracy of optical flow estimation. Thespecific method for constructing the cost volume may be implementedusing the prior art, and detailed descriptions thereof are omittedherein.

In step 403, channel stacking is performed using the CNN feature of theimage of the intermediate frame, the cost volume, bidirectional opticalflow outputted from a previous optical flow estimation, and a CNNfeature of the bidirectional optical flow outputted from the previousoptical flow estimation.

Here, the CNN feature of the bidirectional optical flow outputted fromthe previous optical flow estimation is a CNN feature of the last layerof the optical flow estimation network in the optical flow estimationmodule during the previous optical flow estimation.

In the step, the CNN feature of the image of the intermediate frameobtained in step 401, the cost volume obtained in step 402, thebidirectional optical flow outputted from the previous optical flowestimation, and the CNN feature of the bidirectional optical flowoutputted from the previous optical flow estimation are integrated toachieve recursive bidirectional optical flow estimation.

In one implementation, channel stacking is performed specifically by thefollowing methods.

If the optical flow estimation module is called for the first timecurrently, the channel stacking is performed on the CNN feature of theimage of the intermediate frame, the cost volume, an initialbidirectional optical flow, and a CNN feature of the initialbidirectional optical flow; or otherwise, channel stacking is performedon the CNN feature of the image of the intermediate frame, the costvolume, the bidirectional optical flow outputted from the previousoptical flow estimation by the optical flow estimation module, and theCNN feature of the bidirectional optical flow. The initial bidirectionaloptical flow is 0, and the CNN feature of the initial bidirectionaloptical flow is 0.

In step 404, a channel stacking result is inputted into an optical flowestimation network to perform optical flow estimation, and up-samplingis performed on bidirectional optical flow obtained by the optical flowestimation and a CNN feature of the bidirectional optical flowrespectively, and an up-sampling result is outputted.

Here, considering that the optical flow estimation network needs todown-sample input features when performing optical flow estimation, anoutputted result needs to be up-sampled after optical flow estimation,so as to integrate it with pyramid images participating in the nextoptical flow estimation and corresponding features, or obtainbidirectional optical flow finally matched with target images in scale.

Specifically, in one implementation, if the optical flow estimationmodule is called for the Nth time currently and N is the number oflayers of the image pyramid, a resolution of the up-sampling result ismatched with a resolution of the lowermost image of the image pyramid;or otherwise, a resolution of the up-sampling result is matched with aresolution of an image inputted in the next optical flow estimation.

Referring to FIG. 4 , based on the three-layer image pyramid describedabove, when the optical flow estimation network performs two times ofdouble down-sampling, the optical flow estimated at each layer is ¼ ofthe input image of this layer. Accordingly, double up-sampling isrequired for the outputted results of the first two optical flowestimations, and quadruple up-sampling is required for the outputtedresult of the third optical flow estimation.

The optical flow estimation network used for optical flow estimation inthe optical flow estimation module is a CNN network, which may bespecifically constructed using existing methods. Detailed descriptionsthereof are omitted herein.

It can be seen from the above-mentioned technical solution that in theabove-mentioned method embodiment, bidirectional optical flow estimationis performed in a recursive calling manner based on the image pyramidsfor a target image pair of which optical flow is to be estimated.Moreover, before each optical flow estimation, forward warping towardsmiddle processing is performed firstly on corresponding layers of imagesin the image pyramids, and then bidirectional optical flow estimation isperformed based on an image of the intermediate frame obtained by theprocessing. Thus, by combining the recursive calls of the optical flowestimation module and the image pyramids, the scheme is performed onlyonce, and bidirectional optical flow of the target image pair may beobtained, so that the efficiency, accuracy and generalization ofbidirectional optical flow estimation can be effectively improved, andmodel training and optical flow estimation overheads can be reduced.Therefore, the optical flow estimation scheme proposed in the disclosurecan have a strong application potential under the limitation of highrequirement of real-time performance or low computational powerconsumption for various optical flow-based application scenarios.

Specific applications of the above-mentioned method embodiments areexemplified below in connection with various application scenarios.

FIG. 5 is a schematic diagram of optical flow-based video frameinterpolation according to an embodiment of the disclosure. Referring toFIG. 5 , a frame-0 image and a frame-1 image are inputted, bidirectionaloptical flow between the two frames of images is firstly calculated, andthen pixel synthesis is performed based on the optical flow to obtain anintermediate frame. Referring to FIG. 5 , it can be seen that theoptical flow of small objects may be accurately characterized using themethod embodiment of the disclosure, resulting in better synthesisresults.

FIG. 6 is a schematic diagram of optical flow-based video imagecompletion according to an embodiment of the disclosure. Referring toFIG. 6 , image completion refers to the completion of lost orlow-quality regions in images. Since the optical flow estimation methodadopted in the embodiments of the disclosure can achieve a good balancebetween efficiency and accuracy, real-time video image completion taskscan be served well.

FIG. 7 is a schematic diagram of optical flow-based video salient objectdetection according to an embodiment of the disclosure. Referring toFIG. 7 , a video salient object refers to an object in a video picturethat is most appealing. Referring to FIG. 7 , a moving object in a videois often an arresting object. Optical flow characterization is apixel-level motion, which may be thus regarded as an important promptabout the video salient object. Since the optical flow estimation modeladopted in the embodiments of the disclosure is very lightweight and hashigh optical flow estimation efficiency, the model has a good potentialin real-time salient object detection.

FIG. 8 is a schematic diagram of optical flow-based video objectdetection according to an embodiment of the disclosure. Referring toFIG. 8 , objects of interest in a video tend to have some motion betweenadjacent frames. Optical flow information can help to characterize edgesof moving objects to a large extent, and thus can aid in video objectdetection. The optical flow estimation method adopted in the embodimentsof the disclosure achieves a good balance between speed and accuracy,and thus may have a good potential for application in real-time videoobject detection.

FIG. 9 is a structural schematic diagram of an apparatus according to anembodiment of the disclosure.

Corresponding to the above-mentioned method embodiment, an embodiment ofthe disclosure also proposes a bidirectional optical flow estimationapparatus. Referring to FIG. 9 , the apparatus includes: an imagepyramid construction unit 901, configured to acquire a target image pairof which optical flow is to be estimated, and construct an image pyramidfor each target image in the target image pair; an optical flowestimation unit 902, configured to perform bidirectional optical flowestimation using a pre-trained optical flow estimation model based onthe image pyramid, to obtain bidirectional optical flow between thetarget images, in which an optical flow estimation module in the opticalflow estimation model is recursively called to perform the bidirectionaloptical flow estimation sequentially based on images of respectivelayers in the image pyramid according to a preset order, forward warpingtowards middle processing is performed on an image of a correspondinglayer of the image pyramid before each call of the optical flowestimation module, and an image of an intermediate frame obtained by theforward warping towards middle processing is inputted into the opticalflow estimation module.

It should be noted that the above-mentioned method and apparatusembodiments are based on the same inventive concept. Since theprinciples of the method and the apparatus for solving the problems aresimilar, the apparatus and method implementations may be referred toeach other, and the repetition will be omitted herein.

Based on the above-mentioned bidirectional optical flow estimationmethod embodiment, an embodiment of the disclosure also implements abidirectional optical flow estimation device, including a processor anda memory. The memory stores an application program executable by theprocessor for causing the processor to perform the bidirectional opticalflow estimation method as described above. Specifically, a system orapparatus provided with a storage medium may be provided. Softwareprogram codes realizing the functions of any implementation in theabove-mentioned embodiments are stored on the storage medium, and acomputer (or central processing unit (CPU) or memory protection unit(MPU)) of the system or apparatus is caused to read out and execute theprogram codes stored in the storage medium. In addition, some or all ofactual operations may be completed by an operating system or the likeoperating on the computer through instructions based on the programcodes. It is also possible to write the program codes read out from thestorage medium into a memory arranged in an expansion board insertedinto the computer or into a memory arranged in an expansion unitconnected to the computer. Then, a CPU or the like installed on theexpansion board or the expansion unit is caused to perform some or allof the actual operations through the instructions based on the programcodes, thereby realizing the functions of any of the above-mentionedbidirectional optical flow estimation method implementations.

The memory may be specifically implemented as various storage media suchas an electrically erasable programmable read-only memory (EEPROM), aflash memory, and a programmable program read-only memory (PROM). Theprocessor may be implemented to include one or more central processingunits or one or more field programmable gate arrays that integrate oneor more central processing unit cores. Specifically, the centralprocessing unit or the central processing unit core may be implementedas a CPU or MPU.

An embodiment of the disclosure implements a computer program product,including computer programs/instructions which, when executed by aprocessor, implement the steps of the bidirectional optical flowestimation method as described above.

In an embodiment, a bidirectional optical flow estimation apparatuscomprises: an image pyramid construction unit, configured to: acquire atarget image pair of which optical flow is to be estimated, andconstruct an image pyramid for each target image in the target imagepair, and an optical flow estimation unit, configured to: performbidirectional optical flow estimation using a pre-trained optical flowestimation model based on the image pyramid for each target image toobtain bidirectional optical flow between the target images of thetarget image pair, wherein an optical flow estimation module in theoptical flow estimation model is recursively called to perform thebidirectional optical flow estimation sequentially based on images ofrespective layers in the image pyramid for each target image accordingto a preset order, wherein forward warping towards middle processing isperformed on an image of a corresponding layer of the image pyramidbefore each call of the optical flow estimation module, and wherein animage of an intermediate frame obtained by the forward warping towardsmiddle processing is inputted into the optical flow estimation module.

In an embodiment, a computer program product comprises computer programswhich, when executed by a processor, cause the processor to: acquire atarget image pair of which optical flow is to be estimated, construct animage pyramid for each target image in the target image pair, andperform bidirectional optical flow estimation using a pre-trainedoptical flow estimation model based on the image pyramid for each targetimage to obtain bidirectional optical flow between the target images ofthe target image pair, wherein an optical flow estimation module in theoptical flow estimation model is recursively called to perform thebidirectional optical flow estimation sequentially based on images ofrespective layers in the image pyramid for each target image accordingto a preset order, wherein forward warping towards middle processing isperformed on an image of a corresponding layer of the image pyramidbefore each call of the optical flow estimation module, and wherein animage of an intermediate frame obtained by the forward warping towardsmiddle processing is inputted into the optical flow estimation module.

It should be noted that not all of the steps and modules in the aboveflowcharts and structural diagrams are necessary and that some of thesteps or modules may be omitted according to actual needs. The order ofexecution of the steps is not fixed and may be adjusted as required. Thedivision of various modules is merely to facilitate the description ofthe functional division adopted. In actual implementation, a module maybe divided by multiple modules, the functions of the multiple modulesmay also be realized by the same module, and these modules may belocated in the same device or in different devices.

Hardware modules in the various implementations may be implementedmechanically or electronically. For example, a hardware module mayinclude a specially designed permanent circuit or logic device (e.g. adedicated processor such as FPGA or ASIC) for completing a particularoperation. The hardware module may also include a programmable logicdevice or circuit (e.g. including a general purpose processor or otherprogrammable processors) temporarily configured by software forperforming a particular operation. The implementation of the hardwaremodule using a mechanical mode, or using a dedicated permanent circuit,or using a temporarily configured circuit (e.g. configured by software)may be determined based on cost and time considerations.

As used herein, “schematic” means “serving as an example, instance, orillustration”, and any illustration or implementation described hereinas “schematic” is not to be construed as a preferred or advantageoustechnical solution. For simplicity of the drawings, only those portionsof the drawings that are related to the disclosure are schematicallydepicted in the drawings and do not represent an actual structure as aproduct. In addition, in order to provide a concise understanding of thedrawings, only one of components having the same structure or functionis schematically illustrated in some drawings or may be marked. As usedherein, “a/an” does not represent a limitation of the number of relevantportions of the disclosure to “only one”, and “a/an” does not representthe exclusion of a situation where the number of relevant portions ofthe disclosure is “more than one”. As used herein, “upper”, “lower”,“front”, “back”, “left”, “right”, “inner”, “outer”, and the like areused merely to represent relative positional relationships between therelevant portions and do not define absolute positions of these relevantportions.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. A bidirectional optical flow estimation method,the method comprising: acquiring a target image pair of which opticalflow is to be estimated; constructing an image pyramid for each targetimage in the target image pair; and performing bidirectional opticalflow estimation using a pre-trained optical flow estimation model basedon the image pyramid for each target image to obtain bidirectionaloptical flow between the target images of the target image pair, whereinan optical flow estimation module in the optical flow estimation modelis recursively called to perform the bidirectional optical flowestimation sequentially based on images of respective layers in theimage pyramid for each target image according to a preset order, whereinforward warping towards middle processing is performed on an image of acorresponding layer of the image pyramid before each call of the opticalflow estimation module, and wherein an image of an intermediate frameobtained by the forward warping towards middle processing is inputtedinto the optical flow estimation module.
 2. The method according toclaim 1, wherein the preset order is an ascending order of image scales.3. The method according to claim 1, wherein the performing of theforward warping towards middle processing comprises one of: if theoptical flow estimation module is to be called for a first timecurrently, performing the forward warping towards middle processingbased on uppermost images in the image pyramid for each target image andinitial bidirectional optical flow, to obtain images of intermediateframes corresponding to the uppermost images respectively; or performingthe forward warping towards middle processing based on correspondingimages in the image pyramid of a current call of the optical flowestimation module and bidirectional optical flow outputted from aprevious call of the optical flow estimation module, to obtain images ofintermediate frames of the corresponding images respectively, andwherein the initial bidirectional optical flow is
 0. 4. The methodaccording to claim 1, wherein the performing of the bidirectionaloptical flow estimation by each call of the optical flow estimationmodule comprises: performing feature extraction on the image of theintermediate frame inputted to the optical flow estimation module usinga convolutional neural network (CNN) feature extractor, to obtain a CNNfeature of the image of the intermediate frame; determining acorresponding cost volume based on the CNN feature of the image of theintermediate frame; performing channel stacking using the CNN feature ofthe image of the intermediate frame, the corresponding cost volume,bidirectional optical flow outputted from a previous optical flowestimation, and a CNN feature of the bidirectional optical flowoutputted from the previous optical flow estimation; inputting a channelstacking result into an optical flow estimation network to performoptical flow estimation; performing up-sampling on bidirectional opticalflow obtained by the optical flow estimation and a CNN feature of thebidirectional optical flow respectively; and outputting an up-samplingresult.
 5. The method according to claim 4, wherein the performing ofthe channel stacking comprises one of: if the optical flow estimationmodule is called for a first time currently, performing the channelstacking on the CNN feature of the image of the intermediate frame, thecorresponding cost volume, an initial bidirectional optical flow, and aCNN feature of the initial bidirectional optical flow; or performingchannel stacking on the CNN feature of the image of the intermediateframe, the corresponding cost volume, the bidirectional optical flowoutputted from the previous optical flow estimation by the optical flowestimation module, and the CNN feature of the bidirectional optical flowoutputted from the previous optical flow estimation, and wherein theinitial bidirectional optical flow is 0 and the CNN feature of theinitial bidirectional optical flow is
 0. 6. The method according toclaim 4, further comprising one of: if the optical flow estimationmodule is called for a Nth time currently and N is a number of layers ofthe image pyramid of a target image of the target image pair, matching aresolution of the up-sampling result with a resolution of a lowermostimage of the image pyramid; or matching a resolution of the up-samplingresult with a resolution of an image inputted in a next optical flowestimation.
 7. A bidirectional optical flow estimation devicecomprising: a processor; and a memory, wherein the memory stores anapplication program executable by the processor to cause the processorto: acquire a target image pair of which optical flow is to beestimated, construct an image pyramid for each target image in thetarget image pair, and perform bidirectional optical flow estimationusing a pre-trained optical flow estimation model based on the imagepyramid for each target image to obtain bidirectional optical flowbetween the target images of the target image pair, wherein an opticalflow estimation module in the optical flow estimation model isrecursively called to perform the bidirectional optical flow estimationsequentially based on images of respective layers in the image pyramidfor each target image according to a preset order, wherein forwardwarping towards middle processing is performed on an image of acorresponding layer of the image pyramid before each call of the opticalflow estimation module, and wherein an image of an intermediate frameobtained by the forward warping towards middle processing is inputtedinto the optical flow estimation module.
 8. A non-transitionalcomputer-readable storage medium, storing computer-readable instructionsfor performing a bidirectional optical flow estimation methodcomprising: acquiring a target image pair of which optical flow is to beestimated; constructing an image pyramid for each target image in thetarget image pair; and performing bidirectional optical flow estimationusing a pre-trained optical flow estimation model based on the imagepyramid for each target image, to obtain bidirectional optical flowbetween the target images of the target image pair, wherein an opticalflow estimation module in the optical flow estimation model isrecursively called to perform the bidirectional optical flow estimationsequentially based on images of respective layers in the image pyramidfor each target image according to a preset order, wherein forwardwarping towards middle processing is performed on an image of acorresponding layer of the image pyramid before each call of the opticalflow estimation module, and wherein an image of an intermediate frameobtained by the forward warping towards middle processing is inputtedinto the optical flow estimation module.